Post-rebuttal update: I have read the rebuttal. Thanks for the clarification regarding they type of experiments where there is a larger gap between DAC and the baselines, as well as the clarification on PPO OC/IOPG. The paper proposes a new method for learning options in a hierarchical reinforcement learning set-up. The method works by decomposing the original problem into two MDPs, that can each be solved using conventional policy-based methods. This allows new state-of-the-art methods to easily be'dropped in' to improve HRL.

The paper introduces a double actor critic architecture for learning options. The authors define 2 augmented MDPs for learning the option selection policy as well as the options themselves. Using this MDP formulation, off-the-shelf policy learning algorithms can be used for learning option selection as well as option policies, which was not possible with previous algorithms. The reviews for this paper are borderline. Most reviewers appreciated the intutive idea and the promising results reported in the paper.

Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yielding the Double Actor-Critic (DAC) architecture. Furthermore, we show that, when state-value functions are used as critics, one critic can be expressed in terms of the other, and hence only one critic is necessary. We conduct an empirical study on challenging robot simulation tasks. In a transfer learning setting, DAC outperforms both its hierarchy-free counterpart and previous gradient-based option learning algorithms.

Under this novel formulation, all policy optimization algorithms can be used off the shelf to learn intra-option policies, option termination conditions, and a master policy over options. We apply an actor-critic algorithm on each augmented MDP, yielding the Double Actor-Critic (DAC) architecture. Furthermore, we show that, when state-value functions are used as critics, one critic can be expressed in terms of the other, and hence only one critic is necessary. We conduct an empirical study on challenging robot simulation tasks. In a transfer learning setting, DAC outperforms both its hierarchy-free counterpart and previous gradient-based option learning algorithms.